Skip to content

FINERACT-2485: Idempotency hardening — review follow-ups on #6231 - #6871

Draft
adamsaghy wants to merge 11 commits into
apache:developfrom
adamsaghy:feature/FINERACT-2485
Draft

adamsaghy wants to merge 11 commits into
apache:developfrom
adamsaghy:feature/FINERACT-2485

Conversation

@adamsaghy

@adamsaghy adamsaghy commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Umbrella — not for merging. This draft has been split into nine smaller PRs, one concern each, to make review manageable. Please review and merge those, in this order:

  1. FINERACT-2485: Pass a CommandContext to command hooks (step 1/9) #6907 Pass a CommandContext to command hooks (step 1/9)
  2. FINERACT-2485: Make the Jackson 3 mapper agree with the Jersey wire format (step 2/9) #6908 Make the Jackson 3 mapper agree with the Jersey wire format (step 2/9)
  3. FINERACT-2485: Retry every command handler (step 3/9) #6909 Retry every command handler (step 3/9)
  4. FINERACT-2485: Record what a command acted on and what it did (step 4/9) #6910 Record what a command acted on and what it did (step 4/9)
  5. FINERACT-2485: Claim idempotency keys in the command store (step 5/9) #6911 Claim idempotency keys in the command store (step 5/9)
  6. FINERACT-2485: Make every command result readable from the command store (step 6/9) #6912 Make every command result readable from the command store (step 6/9)
  7. FINERACT-2485: Add idempotency to the new command processing (step 7/9) #6913 Add idempotency to the new command processing (step 7/9)
  8. FINERACT-2485: Refuse a repeat whose claim never finished (step 8/9) #6914 Refuse a repeat whose claim never finished (step 8/9)
  9. FINERACT-2485: Release the idempotency key of a recoverably failed command (step 9/9) #6915 Release the idempotency key of a recoverably failed command (step 9/9)

Each step is one commit on top of the previous one and builds and passes its tests on its own. Every PR targets develop, so it also shows the commits of the earlier steps until they merge; each description links a compare view of that step's own changes.

Relationship to this PR and to #6231

  • The stack carries the code of this PR. Step 9 is identical to this branch's head except for three things: the m_command Liquibase parts are renumbered in step order (0003 entity/action names, 0004 distinctor column and unique key, 0005 response columns; the separate distinctor migration is folded into 0004), a stray comment and blank lines in application.properties are removed, and CustomIdempotencyDistinctorType no longer holds a Supplier field, which error-prone's ImmutableEnumChecker rejects in a full build. That last fix is only in the stack, not on this branch.
  • Steps 1, 5 and 7 carry the hook, module and claim work from FINERACT-2485: Standardize and harden idempotency #6231 by Aleksandar Vidakovic, credited as co-author on those commits. The stack supersedes FINERACT-2485: Standardize and harden idempotency #6231.
  • The first review round on this PR (claim timeout, retry scope and environment variables, no response bodies in logs, changelog numbering) is addressed in the stack.

Known gaps, unchanged

  • Batch API: legacy reads the idempotency key from each batch item's headers; the new hook reads the request header. No batch command reaches a migrated feature yet.
  • With the async or disruptor dispatcher the key is never read, so idempotency is inert (both are off by default).
  • Maker-checker has no counterpart in the new command processing yet.

@adamsaghy
adamsaghy force-pushed the feature/FINERACT-2485 branch 2 times, most recently from d002637 to 983f239 Compare November 9, 2026 23:04

@galovics galovics left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adamsaghy, I know this is a draft on top of #6231 so I'm not treating it as merge-ready, but a few things I'd like us to settle before any of it goes back into #6231.

The replay-from-the-result-object approach makes sense to me, and the startup failure for idempotency without auditing is a good call. My main worry is that the same "425 forever" problem you're guarding against at startup still exists at runtime, plus the retry change reaches way outside idempotency. Details inline.

fineract-command-idempotency/src/main/java/org/apache/fineract/command/idempotency/hook/IdempotencyCheckCommandHook.java:88

+        if (UNDER_PROCESSING.equals(replay.state())) {
+            // the original has not finished, so there is nothing to replay yet and retrying is the right thing to do
+            throw new CommandIdempotencyInProgressException(key, replay.response());
+        }

The PR refuses to start with auditing off because a claimed key would stay UNDER_PROCESSING forever. But the same thing happens at runtime: the pod dies between claim and the after hook, or store(ctx) runs out of retries and goes into the fallback (DLQ file or just a log line). The row stays UNDER_PROCESSING and every retry with that key gets 425 forever, and a client who follows the docs ("retry, it's still running") will retry forever. I don't see any lease/timeout on the claim. Don't we need one, e.g. a claimed-at timestamp and after a configurable period treat the row as abandoned (either let it be re-claimed or answer with a definite error)? I'm not convinced we can ship this without it. Thoughts?

fineract-provider/src/main/resources/application.properties:603

+resilience4j.retry.configs.default.retryExceptions=${FINERACT_RETRY_EXCEPTIONS:org.springframework.dao.TransientDataAccessException,org.springframework.dao.RecoverableDataAccessException,java.sql.SQLTransientException,java.sql.SQLRecoverableException,...}
 ...
-resilience4j.retry.instances.processJobDetailForExecution.retryExceptions=${FINERACT_PROCESS_JOB_DETAIL_RETRY_EXCEPTIONS:...}

Two things here. First, this isn't only the command handlers anymore. processJobDetailForExecution, recalculateInterest and postInterest inherit the new default too, so we're now retrying COB/interest posting on RecoverableDataAccessException / SQLRecoverableException. A connection dropped during commit is exactly the case where we don't know whether the transaction went through, and retrying an interest posting there is how you get it twice. I'd keep the widened list to the command store at least, or leave the recoverable ones out entirely. Second, all the per-instance FINERACT_*_RETRY_EXCEPTIONS env vars are gone. Anyone who set them in their deployment silently loses that config on upgrade. Can we keep them working, or at least call this out as a breaking change in the docs? Also I'd love this to be its own PR, it has very little to do with idempotency.

fineract-command-idempotency/src/main/java/org/apache/fineract/command/idempotency/web/IdempotencyResponseCaptureFilter.java:99

+            log.error("Unable to record the response for idempotency key {} (status {}); the answer was: {}", key, wrapper.getStatus(),
+                    body, e);

Woah, I'd be very careful with this. We're dumping the whole response body into the logs, and same at line 121 for non-JSON answers. These are financial API responses, they can carry client data, and the whole point of the masking work in this PR is to keep that kind of thing out of places it shouldn't be. Let's log the key, status, content type and size, not the body.

fineract-command-jdbc/src/main/resources/db/changelog/tenant/module/command/parts/0005_command_response_cache.xml:25

+    <changeSet author="fineract" id="command-00009" context="postgresql,mysql">
+        <comment>
+            A repeated idempotency key has to be answered exactly as the original request was answered, which means
+            replaying the stored response with the status code it was first sent with.

Minor: the files go 0003 -> 0005 and the IDs go command-00005 -> command-00009. I assume these are leftovers from #6231's history, but let's make them sequential (0004 / command-00006) while nothing has been released. The comment is also stale now, with this follow-up the status code is only stored for failures and non-JSON answers.

vidakovic and others added 7 commits November 12, 2026 14:16
…interface

The hook interfaces now take a CommandContext rather than the command and its result as
separate arguments. WebhookCommandHookAfter and WebhookCommandHookError landed on develop
against the old signatures, so they are adapted here; the after hook still reads the
response before registering the synchronization, so a webhook reports the result as it
was when the command finished.
… wire format

Commands are persisted on Jackson 3 while Jersey can only run on Jackson 2, so the two
mappers have to produce the same bytes. Three Jackson 3 defaults differ and are pinned by
tests: WRITE_DATES_AS_TIMESTAMPS and WRITE_DURATIONS_AS_TIMESTAMPS are off, and
java.time.Month is written as a number with no flag to restore the name - a Jackson 2
client reads that back as an ordinal and gets the wrong month.

It also reapplies credential masking. @JsonSerialize(using = MaskedValueSerializer.class)
is a Jackson 2 annotation and invisible to Jackson 3, so without this external-service
credentials would reach the command table in the clear.
A repeat of a completed command is answered by rehydrating the stored result and handing
it back through the dispatcher, so the resource and the web layer render it exactly as
they rendered the original. Nothing about the original response has to be recorded for
the two to match, which removes the stored status code for successful commands.

The response capture now fills gaps only, enforced in SQL (response IS NULL): the status
and the rendered answer are recorded only for a command that produced no result of its
own. A stored result that cannot be reconstructed is answered 409 stating why, rather
than a 500 or an empty 200, and an answer that cannot be stored is logged so that it stays
recoverable.

Every write to the command row is retried, including claiming the key and recording the
answer. The retry policy is defined once rather than repeated across 47 instances, and
covers everything non-deterministic while ignoring constraint violations explicitly.
Attempt tracking is removed.
The audit hooks own the command row: they move a claimed command out of UNDER_PROCESSING
and store its result. With auditing off, a claimed key would stay UNDER_PROCESSING for
ever and every repeat would be answered 425 - visible only from the second request. An
application configured that way now refuses to start. Both features default to enabled.
A command is recorded as UNDER_PROCESSING from the moment its key is claimed until the audit
hooks record how it finished. If nothing ever does - the process dies in between, or the store
gives up writing the row after its retries - the row stays that way for good, and answering
425 Too Early to every repeat tells the caller to come back for an answer that is never coming.
A caller following the documented advice then retries for ever.

Past fineract.command.idempotency.claim-timeout such a claim is treated as abandoned and the
repeat is refused with 409, saying why. Age is the only signal available: nothing in the row
distinguishes a command that died from one that is merely slow, so the default is generous and
the setting is documented as one to widen rather than tighten.

The key is refused, not released. The command that claimed it may have reached the database
before whatever stopped it, so running it again is the duplicate this mechanism exists to
prevent; establishing whether it took effect is a separate matter, and a retry needs a new key.

No schema change: created_at is written when the key is claimed, and the audit hooks only ever
touch updated_at and executed_at.
@adamsaghy
adamsaghy force-pushed the feature/FINERACT-2485 branch from 983f239 to feb9f2f Compare November 12, 2026 14:42
@adamsaghy

adamsaghy commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review @galovics — all four addressed in the latest push.

Stale claims. You're right, and the startup guard only covered the configuration case, not the runtime one. Added a claim timeout (fineract.command.idempotency.claim-timeout, default 10m): past it, a repeat of a key still recorded as UNDER_PROCESSING is refused with 409 saying the command never finished, rather than 425 for ever.

Deliberately refused and not re-claimed. The command that claimed it may have reached the database before whatever stopped it, so running it again is exactly the duplicate this mechanism exists to prevent; whether it took effect has to be established separately, and a retry needs a new key. Age is the only signal available — nothing in the row distinguishes a dead command from a slow one — so the default is generous and documented as a setting to widen rather than tighten. No schema change was needed: created_at is written when the key is claimed, and the audit hooks only touch updated_at and executed_at.

Retry. Took your suggestion. configs.default is gone, all per-instance retryExceptions lines and their FINERACT_*_RETRY_EXCEPTIONS env vars are restored, and only commandStore carries the wider list, with ignoreExceptions for constraint violations. processJobDetailForExecution, recalculateInterest and postInterest are back on exactly the list they had, so the double-posting risk is gone and nobody loses deployment config on upgrade. The reasoning is in a comment next to the instance: retrying a write of the command row is safe because the idempotency key is unique, which is not true of a commit whose outcome is unknown. What's left is one instance on the idempotency path, so I kept it here rather than splitting it out — happy to pull it into its own PR if you'd still prefer.

Logging the body. Agreed, that was a bad trade — recoverability is not worth putting response bodies in the logs on this API. Both sites now log the key, status, content type and size only. The tests assert the body is absent and the metadata present, so this can't quietly come back.

Numbering. Renumbered to 0004 / command-00006 while nothing is released, and rewrote the stale comment: the status column is now only written for a command that produced no result of its own.

m_command carries approved_at and approved_by_username but nothing that says which entity
and action a command concerns, so the approval columns cannot be used: maker-checker derives
the permission governing approval by joining the two names with _CHECKER, the way the legacy
processing does. The audit search and the checker inbox are defined in the same terms - the
inbox returns the actionNames and entityNames a requestor may approve - so a row without them
is invisible to both. This adds entity_name and action_name, and the means of filling them.

Declared as abstract methods on Command rather than an optional interface. The interface shape
is what left 44 of 47 commands without names; a command that forgets them would be stored with
none, and a row no audit or inbox can find is worse than a compile error. The columns are
therefore NOT NULL, with rows written before the change backfilled as UNKNOWN.

The names are the ones in m_permission, checked against the 953 pairs the seed data defines,
and they are not always what the class name suggests: CacheSwitchCommand is CACHE/UPDATE,
the entity is BUSINESS_DATE rather than BUSINESSDATE, collateral is COLLATERAL_PRODUCT and
CLIENT_COLLATERAL_PRODUCT, chart slabs are CHARTSLAB, and attendance is an action on MEETING.
Notes and images act on different entities depending on the request - a note belongs to
whatever it is attached to, an image to a client or a member of staff - so those compute the
name from the payload, which is set before the command is stored.

Two consequences worth naming. HookableCommand loses its two methods and becomes a marker:
they duplicated what every command now declares, and it only ever meant "notify webhooks".
And an abstract Command cannot be the target of the mapper's read direction, so a row read
back becomes a StoredCommand carrying the recorded names - there is no class left to ask.

The migration test rolls back a fixed number of changesets, which has to grow with every part
added after 0003, or it leaves the unique constraint in place and the re-seed collides.
…lay gaps

An idempotency key is now unique together with a distinctor, stored beside it in the new
NOT NULL column m_command.custom_idempotency_distinctor (default ''). The composite unique
constraint replaces the key-only one; it is added before the old one is dropped, so the key
is never left unconstrained. The configured CustomIdempotencyDistinctor computes the value
when the key is claimed: ENTITY_NAME_AND_ACTION_NAME (the default, e.g. LOAN|CREATE) scopes
a key the way the legacy processing does, NONE makes it unique per tenant. Every lookup by
key - claim, replay, request comparison, recorded response - also matches the distinctor,
and IdempotencyKeyLookupCoverageTest fails the build on a lookup that does not.

Review fixes on the same path:

- Four collateral command results had no constructor Jackson could use, so a repeat of
  those requests was refused with 409 instead of replayed. They are now constructible, and
  CommandHandlerResultRehydrationTest reads every handler's result type back the way a
  replay does. The 409 for a result that cannot be rebuilt no longer invites a retry under
  a new key: the command did take effect.
- Idempotency refuses to start with the audit after or error hook switched off, not only
  with auditing off: without them a command never leaves UNDER_PROCESSING.
- A claim is always stored with its time, even with the timestamp hook off, so the claim
  timeout can tell an abandoned claim from a running one.
…mmand

The legacy command processing lets a retry under the same key execute again after a
recoverable failure (FINERACT-2985): an exception marked RecoverableError, or one whose error
codes are all in the idempotency-recoverable-error-codes global configuration. The new command
processing replayed every failure for ever, so a client retrying a transient failure on a
migrated endpoint could not get past it.

A command that fails recoverably now moves its record to <key>_RE<id>, the scheme the legacy
processing uses, hashing a key too long to take the suffix. The original key is free, so the
retry is executed as a new command, and the failure stays on record under the moved key with
the answer the client was sent. If the moved key is taken or the move fails, the failure stays
under the original key, as for any other failure.

The decision sits behind RecoverableCommandErrorClassifier in fineract-command, so the command
modules stay independent of fineract-core; fineract-core supplies an adapter that delegates to
the legacy RecoverableErrorClassifier. Without a classifier, nothing is recoverable. The key is
moved on the command before the audit error hook runs, because that hook rewrites the whole row
from the command.
The legacy command processing answers a repeat of an unfinished command 425 Too Early for as
long as it stays unfinished. The new command processing turns that into 409 after
fineract.command.idempotency.claim-timeout, so that a claim abandoned by a dead process does
not answer 425 for ever.

The timeout stays on by default, at 10 minutes. Setting it to 0, or leaving it empty, now
switches it off and restores the legacy answer, for deployments whose clients rely on it. A
negative value is refused at startup: a negative age is always exceeded, so even a command
that is still running would be declared abandoned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants